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1.  INTRODUCTION 


This  study  evaluates  the  skill  of  a  forecasting  method*  that 
utilizes  surface-observed  meteorological  variables  to  develop  short 
range  (3-12  hr)  predictions  of  evaporation  duct  heights.  Although 
this  method  was  developed  for  use  in  short  range  statistical 
terminal  forecasting,  its  adaptation  to  duct  height  prediction 
required  only  the  selection  of  appropriate  predictors  and  individual 
internal  transformations. 

Data  from  two  ocean  weather  stations  (OWS)  in  the  North 
Atlantic  Ocean,  OWS  Charlie  and  OWS  Echo,  were  used  in  the  evalua¬ 
tion,  the  former  to  represent  the  higher  latitudes  and  the  latter 
to  represent  the  midlatitudes.  For  purposes  of  this  evaluation, 
skill  and  threat  scores  derived  from  these  data  were  compared  to 
persistence  scores. 

The  forecast  method,  the  OWS  data  samples,  and  the  evaluation 
procedure  are  described  in  Section  2.  Results  are  discussed  in 
Section  3.  Threat  and  skill  scores  are  defined  in  the  Appendix. 


* 

Developed  by  Dr.  R.  G.  Miller,  Techniques  Development  Laboratory, 
NOAA,  Silver  Spring,  MD  20910. 
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2.  FORECAST  METHOD 


2.1  INTRODUCTION 

Forecasters  presently  use  a  combination  of  climatology, 
persistence,  and  data  on  the  existing  synoptic  situation,  to  develop 
short  range  (1-12  hr)  weather  predictions.  Indications  of  the 
existing  synoptic  situation,  in  turn,  can  be  found  in  certain 
meteorological  variables  observed  at  the  surface  every  one  to  three 
hours. 

Using  this  premise  that  surface  observables  can  provide  valid 
synoptic  predictors.  Miller  (1979)  designed  a  conditional  climato¬ 
logical  prediction  method  for  short  range  forecasting  based  on  a 
technique  called  regression  estimate  of  event  probability  (REEP). 
Applying  the  basic  Markovian  process  to  the  REEP,  Miller  expanded 
the  technique  to  develop  hourly  forecasts  for  one  to  twenty -four 
hours;  these  forecasts  are  the  event  probabilities  determined  by 
the  regression  equations,  converted  to  categorical  forecasts  by  a 
set  of  threshold  probabilities. 

It  was  reasonable  to  extend  this  method  to  the  prediction  of 
evaporation  duct  heights,  insofar  as  these  ducts  are  surface  layer 
phenomena . 

2.2  DISCUSSION 

As  an  example,  consider  attempting  to  forecast  winds  and 
pressure  using  only  these  same  variables  as  predictors.  Wind  speed 
(WS),  wind  direction  (WD)  and  sea  level  pressure  (P)  are  transformed 
into  (0,1)  dummy  variables  where  the  value  is  set  equal  to  1  if  the 
observation  falls  within  the  dummy's  prescribed  interval,  and  to  0 
otherwise.  A  possible  transformation  would  be  the  following: 


See  Miller,  1964. 


FHLChDItvO  FACS  BUML-NOT  F1LMWD 
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P1  (50-990),  P2  (990-1000),  P3  (1000-1010) 

P4  (1010-1015),  P5  (1  01  5-1  020),  P6  (1  020-1  1  00) 

WS  (kt)  WS 1  (0-4),  WS2  (4-8),  WS3  (8-12) 

WS4  (12-20),  WS5  (20-30),  WSg ( 30-1 00 ) 

WO  WD-j  (N-E),  WD2  (E-S),  WD3  (S-W),  W04  (W-N) 

These  three  variables  are  then  transformed  into  16  (0,1) 
dummy  variables;  if  the  observed  P  is  1013,  then  p4  =  1 ,  and  for  all 
other  i's,  P.=0.  Wind  speed  and  direction  are  set  in  like  manner. 

A  regression  equation  is  then  calculated  for  each  of  the  dummy 
variables. 

Using  all  16  dummies  as  predictors.  Miller  (1979)  has  shown 
that  this  method  gives  probabilities  directly  from  the  regression 
equation,  such  that 

PCPyl]  =  aQ  +  1?  Vr  (2’]) 

p C P 1 = 1  ]  is  the  probability  that  p1  (i.e.,  pressure  is  between 
950  and  990)  equals  one;  a^  are  the  regression  coefficients,  and 
Z  =  (0010  ....  0)  is  a  vector  whose  components  are  1  if  the 
observed  P,  WS,  and  WD  occur  with  the  given  range  and  are  0  if  not. 
There  are  16  such  regression  equations, 

p[ p i =i  3  =  v  + ;  vt 


P[pl6  =  l  ]  -  a0i]6  +  Z  a.  j6zi 


4 


Variabl e 
P  (mb) 


The  coefficients  can  be  represented  by  a  16x16  matrix  0. 


so  that 


0  = 


0,1 

al  ,1  al  ,2  •  • 

•  •  al  ,1  6 

0,2 

al  ,2  a2 ,2  '  ‘ 

•  ‘  a2  ,1  6 

V3  0 , 1  6  a  1  , 1  6 


.  a 


16,16 


£  =  Z/Q_,  where 

P  -  (P0P1  •  •  •  P16) 


(2-4) 

(2-5) 


Pg  =  1 ,  since  it  is  simply  a  constant  needed  for  regression; 
p  ^  i  =  1 »  •  ••,  16)  are  the  probabilities  that  the  respective  dummy 
variables  will  be  1.  If  the  regression  coefficients  are  determined 
from  hourly  observations,  the  predicted  values  will  be  for  one  hour 
later.  To  generate  a  n-hour  forecast,  the  first  order  Markov 
process  is  assumed,  and  one  has  P ( n )  =  Z  0 n . 

A  true  first  order  Markov  process  is  one  in  which  the  next 
event  depends  only  on  the  preceding  event  and  not  on  the  earlier 
chain  of  events.  Whether  a  Markov  first  order  process  is  applicable 
to  meteorological  phenomena  depends  on  the  extent  to  which  the  real 
world  violates  the  simple  model.  This  model  has  been  shown  to  hold 
fairly  well  (Miller,  1979).  Empirical  tests  such  as  the  one 
reported  here  are  the  only  way  to  examine,  indirectly,  the  validity 
of  the  Markov  assumption. 

The  advantages  and  disadvantages  of  transforming  continuous 
variables  such  as  temperature  into  discrete  variables  (the  (0,1) 
dummies)  are  summarized  as  follows: 


Advantages 

(1)  Probabi 1 i ti es  computed  directly  from  regression 
equations. 

(2)  Facilitate  computations 

(3)  Probabilities  sum  to  zero. 

(4)  Probabilities  essentially  minimize  the  Brier-Alien 
P  score. 


*See  Miller,  1964. 
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Pi sadvantaqes 

(1)  Probabilities  are  not  bounded  by  0  and  1. 

(2)  Transformation  from  a  continuous  variable  to  a 

discrete  variable  discards  some  information. 

The  first  disadvantage  can  be  corrected  by  normalizing  those 
forecast  probabilities  that  do  exceed  the  (0,1)  interval. 

The  second  disadvantage  is  accepted  as  a  reasonable  trade-off 
in  order  to  handle  the  Markov  process  by  simple  matrix  algebra. 

This  is  accomplished  by  using  the  vectors  of  dummy  variables, 
rather  than  the  "true  state”  vector.  In  this  sense  the  process 
is  an  "equivalent  Markov  method."  It  would  be  impractical, 
even  on  large  computers,  to  attempt  to  compute  the  true  Markov 
transition  matrix  for  all  states,  using  even  a  small  number  of 
predictors  (see  Whiton,  1978). 

Once  the  forecast  probabilities  for  each  dummy  variable  are 
computed,  the  question  arises  of  how  best  to  use  these  probabilities 
to  generate  a  forecast.  Miller  and  Best  (1979)  have  developed  a 
method  to  determine  statistically  appropriate  threshold  probabili¬ 
ties,  P*,  so  that  a  set  of  categorical  forecasts  C  can  be  found  by 

C  =  P-P*  (2-6) 

The  C's  are  either  zero  or  one,  indicating  which  dummy  is 
categorically  forecasted. 

2.3  EVAPORATION  DUCT  HEIGHT  FORECASTING 

The  adaptation  of  the  "equivalent  Markov  method"  to  the 
problem  of  forecasting  evaporation  duct  heights  requires  first  the 
selection  of  appropriate  meteorological  variables  from  those 
recorded  in  the  marine  surface  observations  data  file.  Duct  height 
is  calculated  from  the  air,  sea  and  dew  point  temperatures,  and 
wind  speed  (Jeske,  1971;  Hitney,  1975).  The  calculated  duct  height 
is  then  added  to  the  selected  meteorological  variables  to  form  the 
data  base  from  which  the  regression  coefficients  are  calculated. 


*See  Miller,  1964. 

il 
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The  variables  selected  to  form  the  predictor  set  for  this 
cresent  study  are  listed  in  Table  1.  The  weather  types  are  simply 
the  occurrences  or  non-occurrences  of  the  variable.  These  19 
variables  were  then  transformed  through  interval  t ra ns fo rma t i on  to 
the  111  dummy  variables  listed  in  Table  2  for  ocean  weather  station 
Echo. 

Table  1.  Variables  in  the  predictor  set. 

1 .  Month 

2 .  Time 

3.  Wind  direction 

4 .  Wind  speed 

5 .  Air  temoerature 

6 .  Dew  point 

7.  Sea  surface  temperature 

3.  Air-sea  temperature  difference 

9.  Sea  level  pressure 

10.  Pressure  change  (3  m) 

11.  Total  cloud  amount 

12.  Lower  cloud  amount 

13.  Duct  height  -  present  weather 

14.  Rain,  drizzle,  snow 

15.  Rain  showers,  snow  showers 

1 6 .  Fog  -  past  weather 

17.  Rain,  drizzle,  snow 

18.  Rain  showers,  snow  showers 

19.  Fog 


Table  2 


W i  nd 


Wind 


Month 

T  i  me 

Di  recti  on 

Speed 

Jan 

(1) 

2000-0900 

(13) 

N-NE  (16) 

0- 

8 

(24) 

Feb 

(2) 

0900-1300 

(14) 

NE- 

E  (17) 

9- 

1  1 

(25) 

Ma  r 

(3) 

1 300-2000 

(15) 

E-SE  (18) 

1  2- 

1  4 

(26) 

Apr 

(4) 

SE- 

S  (19) 

1  5- 

1  7 

(27) 

May 

(5) 

S-SW  (20) 

18- 

20 

(28) 

Jun 

(6) 

SW- 

W  (21) 

21  - 

23 

(29) 

Jul 

(7) 

W-NW  (22) 

24- 

27 

(30) 

Aug 

(8) 

NW- 

■N  (23) 

28- 

■30 

(31  ) 

Sep 

(9) 

31  - 

•99 

(32) 

Oct 

(10) 

Nov 

(ID 

Dec 

(12) 

Ai  r 

Sea 

Surface 

Temperature 

Dew 

Point 

Temperature 

-30  . 

-15. 

6 

(33) 

-30. 

- 

8 . 5 

(43) 

- 

.  1  - 

17.0  (54) 

1  5  . 

,7  -16. 

,7 

(34) 

8  . 

,6- 

11.0 

(44) 

17. 

1  - 

17.8  (55) 

16. 

.8  -17. 

,8 

(35) 

11  . 

1  - 

12.5 

(45) 

1  7  . 

.9- 

18.3  (56) 

1  7  . 

.9  -18. 

.5 

(36) 

12. 

,6- 

13.5 

(46) 

18. 

,4- 

19.5  (57) 

18, 

.6  -19. 

.2 

(37) 

13, 

.6- 

15.5 

(47) 

19, 

.6- 

21.0  (58) 

19, 

.2  -20, 

.0 

(38) 

15 

.6- 

16.5 

(48) 

21  , 

.1  - 

22.7  (59) 

20, 

.  1  -21 

.  7 

(39) 

16 

.6- 

17.5 

(49) 

22. 

.8- 

23.9  (60) 

21 

.8  -24 

.0 

(40) 

1  7 

.6- 

18.5 

(50) 

24 

.0- 

25.6  (61 ) 

24 

.1  -25 

.  5 

(41) 

18 

.6- 

20.5 

(51  ) 

25 

.6- 

99  (62) 

25 

.6  -  9 

.9 

(42) 

20 

.6- 

•21  .5 

(52) 

21 

.6- 

■99 

(53) 
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Table  2  (continued) 


Sea  Level 


TA-TS 

Pressure 

Pressure 

Cha  nq 

-30  .-4.5 

(63) 

800-1 01 2 

(71  ) 

-  90-1.5 

(78) 

1 

4^ 

4^ 

I 

ro 

CD 

(64) 

101 2-101 7 

(72) 

-1  .4-1  .2 

(79) 

-2.7-1  .6 

(65) 

1017-1020 

(73) 

-1  .1  -0.9 

(80) 

-1  .5-1  .0 

(66) 

1020-1022 

(74) 

-0 . 8-0 . 5 

(81  ) 

-0.9-0. 5 

(67) 

1 022-1024 

(75) 

-0.4-0. 2 

(82) 

O 

4-* 

I 

O 

O 

(  68  ) 

1 024-1027 

(76) 

-0.1  -0.0 

(83) 

0.1 -0.8 

(69) 

1027-1 100 

(77) 

0.1 -0.3 

(84) 

0.8-99 

(70) 

0.4-0. 8 

(85) 

0.9-1  .5 

(86) 

1 .6-99 

(87) 

Total  Cloud  Lower  Cloud 


Amount 

Amo  u  n  t 

Duct 

H  e  i  g  h  t 

Clear 

(38) 

Clear 

(92) 

0-8 

(96) 

Scattered 

(89) 

Scattered 

(93) 

9-12 

(97) 

Broken 

(90) 

Broken 

(94) 

1  3-20 

(98) 

Overcast 

(91  ) 

Overcas  t 

(95) 

21  -90 

(99) 

Present  Weather 

Past  Weather 

R.L.S 

(100) 

R.L.S 

(106) 

No  R.L.S 

(101  ) 

No  R.L.S 

(107) 

RW  ,  SW 

(102) 

RW ,  SW 

(108) 

No  RW  , SW 

(103) 

No  RW  , SW 

(109) 

Fog 

(104) 

Fog 

(HO) 

No  Fog 

(105) 

No  Fog 

(111) 
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The  intervals  chosen  were  such  that  the  relative  frequencies 
of  occurrence  were  roughly  the  same  (within  a  factor  of  2),  with 
the  exception  of  duct  height  intervals  which  were  selected  on  the 
basis  of  operational  factors.  Those  intervals  for  variables  such 
as  temperature  are  1  a t i t ude- 1 ong i tude  dependent;  OWS  Charlie 
therefore  has  intervals  somewhat  different  than  those  listed  in 
Table  2  for  OWS  Echo.  The  total  number  of  dummies  in  each  case  is 
the  same. 

The  regression  coefficients  for  the  equations  for  each  of 
these  dummy  variables  were  calculated  using  a  10-year  data  base 
with  observations  at  three-hour  intervals.  The  probabilities  and 
categorical  forecasts  were  then  obtained  by  matrix  multiplication, 
as  shown  in  Eqs.  2-4  and  2-6. 

2.4  DATA  BASE 

Surface  observations  for  OWS  Charlie  (52°N,  35°W)  and  OWS  Echo 
(35°N,  48 °W )  were  used  to  evaluate  the  forecast  procedure.  Ten-year 
data  samples  for  each  OWS  were  used  to  calculate  the  regression 
coefficients.  Independent  data  samples  of  about  three  years  for 
each  station  were  also  processed  to  be  used  to  produce  forecasts 
and  skill  scores.  Each  meteorological  variable  was  screened  for 
reasonable  values,  and  observations  with  any  missing  variables  were 
discarded. 

2.5  EVALUATION  PROCEDURE 

Evaporation  duct  height  fo recasts  for  three,  six,  nine,  and 
twelve  hours  were  produced,  and  then  compared  to  the  observed  duct 
heights.  Heidke  skill  scores,  threat  scores  and  total  percent 
correct  were  computed.  Since  persistence  is  the  best  objective 
forecast  technique  currently  available,  it  was  also  tabulated  and 
skill  scores  computed 

The  scores  were  subjected  to  a  one-way  analysis  of  variance 
to  determine  statistically  significant  differences  in  the 
equivalent  Markov  method  and  persistence  forecast  skills. 
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3.  RESULTS 


Contingency  tables  were  constructed  so  that  the  columns 
contain  the  forecasts  and  the  rows  contain  the  observed  values  of 
duct  heights.  Tables  3  through  18  give  the  cell  counts  and  scores 
for  both  ocean  weather  stations.  The  "a"  tables  are  the  various 
forecasts  using  the  equivalent  Markov  forecast  procedure;  the  "b1' 
tables  are  those  for  persistence.  Both  dependent  and  independent 
sample  results  are  shown. 

The  threat  score  for  each  diagonal  cell  and  the  weighted 
average  for  the  threat  scores  were  computed  (see  Appendix); 

Heidke  skill  scores  for  each  contingency  table  also  were  computed. 

3.1  OWS  CHARLIE 

The  Heidke  scores  for  both  independent  and  dependent  data 
samples  in  Table  19a  show  that  the  forecast  procedure  (F)  appears 
better  overall  than  persistence  (P).  The  averages  over  all  four 
forecast  periods  are  also  larger  for  F  than  for  P,  for  both  data 
samples.  An  analysis  of  variance  was  performed  on  both  samples; 
the  F-values  and  pro ba b i 1 i t i es  are  given  in  Tables  19a  and  19b. 

The  analysis  answers  the  question  of  whether  the  forecast 
average  scores  really  are  higher  than  the  persistence  average 
scores.  For  OWS  Charlie,  statistically  there  is  a  32  %  (61  'i )  chance 
for  the  "no"  answer  for  the  independent  (dependent)  sample  (see 
Table  19a).  Therefore  one  must  reject  that  answer  and  accept  that 
the  average  scores  for  the  forecast  procedure  and  simple  persistence 
are  not  s ta t i s t i ca 1 1 y  different.  Hence  the  forecast  procedure  is 
no  better  than  persistence,  based  on  the  Heidke  scores. 

Threat  scores  were  also  subjected  to  an  analysis  of  variance. 
Table  19a  shows  that  the  average  threat  score  for  the  independent 
sample  are  .579  and  .547  for  F  and  P,  respectively.  The  analysis 
shows  no  statistical  difference  between  these  averages  ( 44 Z  chance 
of  being  wrong  if  one  assumes  they  are  different).  The  dependent 
sample  shows  even  stronger  evidence  that  the  forecast  procedure  is 
no  better  than  persistence. 


*The  necessary  assumption  of  similar  sample  variances  is  clearly 
satisfied  (see  Table  17). 

11 


3.2  OWS  ECHO 


Heidke  scores  are  shown  in  Table  19b  for  both  data  samples 
for  OWS  Echo.  An  analysis  of  variance  strongly  indicated  that  the 
forecast  procedure  was  no  better  than  persistence.  The  threat 
scores  shown  in  Table  19b  also  reveal  small  F-test  values,  and  high 
probabilities  that  these  two  set  of  scores  are  indeed  similar. 

3.3  DISCUSSION  OF  RESULTS 

The  forecast  method  was  compared  to  persistence  via  Heidke 
skill  scores  and  threat  scores.  Comparisons  were  made  for  both 
the  dependent  data  sample  (the  data  used  to  generate  the  forecast 
equations)  and  an  independent  data  sample.  Data  from  OWS  Charlie 
(a  northern  latitude  location)  and  OWS  Echo  (a  midlatitude 
location)  were  used  to  examine  the  forecast  skills  for  different 
evaporation  duct  height  distribution  (see  Sweet,  1979;  and  Tables 
1-16  of  this  report). 

The  northern  latitude  station,  Charlie,  had  a  duct  height 
distribution  skewed  toward  the  lowest  duct  height  cell  (0-8  m). 

The  Heidke  skill  scores  were  higher  for  this  station  than  for  Echo, 
whose  distribution  was  centered  about  the  12-20  m  cell.  The  same 
difference  was  noted  for  the  threat  scores  (see  Tables  19a  and  19b). 
Since  the  higher  duct  heights  affect  more  fleet  radar  systems, 

Echo's  duct  height  distribution  was  representative  of  a  more 
operationally  significant  distribution  type.  Therefore,  lessor 
skills  of  Echo  as  compared  to  Charlie  is  discouraging.  This  trend 
was  also  revealed  in  the  total  percent  of  correct  forecasts  shown 
in  Tables  20a  and  20b. 

An  analysis  of  variance  comparing  the  forecast  method's  skills 
to  the  persistence  skills  revealed  that  the  forecast  method  was  not 
statistically  better  than  simple  persistence.  Some  forecast - va 1 i d 
periods  (i.e.,  Echo's  Heidke  skill  for  the  six  hour  forecast)  taken 
singularly  appeared  to  show  that  the  forecast  method  was 
measureably  better  than  persistence.  However,  when  both  Heidke 
and  threat  scores  were  considered,  no  forecast-val i d  period 
consistently  had  higher  scores  than  persistence. 
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It  is  concluded  tnat  the  forecast  algorithm  as  a  predictive 
met nod  is  no  better  than  simply  forecasting  the  continuation  of 
current  observed  duct  heights.  It  appears  that  further  study  or 
development  of  the  forecast  method  evaluated  here  would  be 
unproductive  unless  new  developments  of  the  method  promise  greater 
skill. 


Table  3.  OWS  Charlie  Independent  Sample. 


OBSERVED 


OBSERVATION 


3  HR  FORECAST 
FORECAST 


0-8 

0^' 

8-1  2 

1  2-20 

>20 

TOTAL 

0-8 

2958 

506 

21 

10 

3495 

8-12 

31  2 

849 

80 

7 

1  248 

12-20 

29 

1  34 

50 

6 

21  9 

o 

CM 

A 

1  3 

1  3 

4 

8 

38 

TOTAL 

331  2 

1  502 

1  55 

31 

5000 

a 


CORRECT 

= 

771 

SKILL 

a 

.507 

THREAT 

(0-8) 

3 

.762 

(8-12) 

a 

.446 

(12-20) 

a 

.  1  54 

(  >20) 

* 

.1  31 

Average 

3 

.652 

3  HR  PERSISTENCE 
FORECAST 


0-8 

* 

8-12 

12-20 

>20 

TOTAL 

0-8 

2865 

384 

41 

11 

— 

3301 

8-12 

421 

690 

152 

6 

1  269 

1  2-20 

44 

1  56 

1  58 

1  0 

368 

>20 

1  2 

1  6 

1  3 

21 

62 

TOTAL 

3342 

1246 

364 

48 

5000 

h 


CORRECT  *  750 
SKILL  =  .423 
THREAT 

(0-8)  *  .753 

(8-12)  *  -373 

(12-20)  *  -275 
( >20)  *  -235 

Average  *  -621 


Table  4.  9 US  Charlie  Independent  Sample. 


OBSERVED 


OBSERVATION 


6  HR  FORECAST 
FORECAST 


..V? 

0-8 

00 

1 

o 

eg 

i 

CO 

1  2-20 

>20 

TOTAL 

947 

291 

■ 

0 

1  250 

00 

I 

ro 

151 

451 

41 

■ 

12-20 

18 

69 

■ 

0 

99 

o 

CVJ 

A 

6 

0 

0 

7 

TOTAL 

1  1  22 

81  2 

65 

■ 

2000 

CORRECT  =  70S 
SKILL  ■  .429 
THREAT 

(0-8)  =  -664 

(8-12)  *  -448 

(12-20)  =  -973 
( >20)  =  .0 
Average  *  -569 


6  HR  PERSISTENCE 
FORECAST 

V^O-8  8-1  2  1  2-20  >20  TOTAL 

0-8 

8-1  2 
1  2-20 
>20 
TOTAL 


1569 

258 

37 

8 

1872 

306 

327 

35 

0 

718 

51 

91 

81 

4 

227 

1  8 

7 

6 

• 

0 

31 

1  944 

683 

209 

1  2 

2848 

b 


CORRECT 

*  6  9  "> 

SKILL 

*  .369 

THREAT 

- 

(0-8) 

=  .712 

(8-12) 

=  .304 

(12-20) 

=  .228 

( >20) 

=  .0 

Average 

*  .569 
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Table  5.  OWS  Charlie  Independent  Sample. 


9  HR  FORECAST 
FORECAST 

0-8  8-12  12-20  >20  TOTAL 

0-8 


8-12 

OBSERVED 

1  2-20 


>20 


TOTAL 


999 

297 

4 

0 

1  300 

173 

388 

21 

0 

582 

30 

66 

8 

0 

1  04 

10 

4 

0 

0 

14 

1  21  2 

755 

33 

0 

2000 

a 


9  HR  PERSISTENCE 
FORECAST 

0-8  8-12  12-20  >20  TOTAL 

0-8 

8-12 

OBSERVATION 

12-20 
>20 
TOTAL 


779 

1  44 

26 

0 

949 

177 

1  60 

48 

2 

387 

43 

47 

31 

0 

121 

12 

■ 

3 

B 

20 

101  1 

355 

108 

3 

1477 

h 


CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20 ) 
Average 


CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


Table  7.  OWS  Charlie  Dependent  Sample. 


OBSERVED 


OBSERVATION 


3  HR  FORECAST 
FORECAST 


f- 

o 

• 

00 

8-1  2 

12-20 

>20 

TOTAL 

0-8 

861 

292 

9 

5 

1167 

8-12 

1  25 

484 

54 

2 

665 

12-20 

16 

93 

46 

2 

157 

>20 

4 

3 

2 

2 

1  1 

TOTAL 

1006 

872 

1  1  1 

11 

2000 

a 


CORRECT  *  7 

SKILL  *  .4 

THREAT 

(0-8)  *  .6 
(8-12)  *  .4 

(12-20)  *  .2 
( >20)  *  -1 
Average  *  .5 


3  HR  PERSISTENCE 
FORECAST 


.^0-8 

8-12 

1  2-20 

>20 

TOTAL 

V 

0-8 

1043 

164 

26 

6 

1  239 

8-1  2 

1  68 

291 

78 

4 

541 

1  2-20 

24 

82 

84 

7 

197 

>20 

Bj 

6 

6 

23 

TOTAL 

1  242 

541 

194 

23 

2000 

CORRECT  *  7 

SKILL  *  .4 

THREAT 

(0-8)  -  .7 

(8-12)  *  .3 

(12-20)  =  .2 
( >20)  *  -1 
Average  *  -5 


Table  8.  OWS  Charlie  Dependent  Sample. 


OBSERVED 


OBSERVATION 


6  HR  FORECAST 
FORECAST 


£ 

o 

1 

Oo 

8-12 

1  2-20 

>20 

TOTAL 

0-8 

848 

307 

1  6 

1 

1  1  72 

8-12 

152 

464 

41 

2 

659 

1  2-20 

21 

1  05 

31 

0 

1  57 

>20 

9 

3 

0 

0 

1  2 

TOTAL 

1030 

879 

88 

3 

2000 

a 


CORRECT  =  671S 

SKILL  »  .403 
THREAT 

(0-8)  =  .626 
(8-12)  =  -432 

(12-20)  =  .145 
( >20)  =  -000 
Average  =  -519 


6  HR  PERSISTENCE 
FORECAST 


-  0-8 

C\J 

1 

00 

1  2-20 

>20 

TOTAL 

0-8 

965 

188 

48 

1  1 

1  21  2 

C\J 

1 

.00 

1  88 

250 

-  - 

88 

8 

534 

12-20 

54 

82 

77 

7 

220 

>20 

10 

1  6 

5 

3 

34 

TOTAL 

1  21  7 

536 

218 

29 

2000 

h 


CORRECT  =  65% 

SKILL 


THREAT 

(0-8)  =  .vjjj 

(8-12)  =  -304 

(12-20)  =  .213 
( >20)  =  -050 

Average  *  -505 


Table  9.  OWS  Charlie  Dependent  Sample. 


OBSERVED 


OBSERVATION 


9  HR  FORECAST 


FORECAST  , 


0-8 

8-1  2 

O 

CM 

1 

CM 

>20 

TOTAL 

V 

0-8 

81  9 

329 

8 

0 

1 1  56 

8-12 

197 

461 

24 

0 

682 

1  2-20 

33 

1  07 

1  0 

0 

1  50 

>20 

9 

3 

0 

0 

1  2 

TOTAL 

1058 

900 

42 

0 

2000 

CORRECT 

6  5*. 

SKILL 

= 

.342 

THREAT 

(0-8) 

3 

.587 

(8-12) 

x 

.411 

(12-20) 

= 

.055 

( >20) 

3 

.000 

Average 

3 

.490 

a 


9  HR  PERSISTENCE 
FORECAST 


v/'  0-8  8-12  12-20  >20  TOTAL 

0-8 

8-1  2 
12-20 
>20 

TOTAL 


918 

1  99 

62 

-  - 

1 1 

1  1  90 

186 

220 

1 1 1 

9 

_ 

526 

81 

95 

75 

4 

255 

1  1 

I  1 

3 

4 

--  J 

29 

1196 

525 

251 

28 

2000 

CORRECT 

3 

61  X 

SKILL 

* 

.  300 

THREAT 

(0-8) 

3 

.625 

(8-12) 

3 

.265 

(12-20) 

3 

.  1  74 

( >20 ) 

3 

.075 

Average 

X 

.465 

Table  10.  OWS  Charlie  Dependent  Sample. 


12  HR  FORECAST 
FORECAST 

^  0-8  8-12  12-20  >20  TOTAL 


OBSERVED 


832 

322 

5 

0 

1  1  59 

234 

420 

9 

0 

663 

50 

1  06 

4 

0 

1  60 

1  3 

5 

0 

0 

1  8 

1  1  29 

853 

18 

0 

2000 

CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


12  HR  PERSISTENCE 
FORECAST 


OBSERVATION 


V? 

^  0-8 

8-1  2 

1  2-20 

>20 

TOTAL 

V 

0-8 

901 

223 

66 

1  1 

1  201 

8-1  2 

209 

225 

99 

4 

- 

537 

1  2-20 

75 

98 

64 

3 

240 

CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


0  22 


1196 


553 


18 


2000 


Table  11.  OWS  Echo  Independent  Sample. 


3  HR  FORECAST 
FORECAST 

.V^O-8  8-1  2  1  2-20  >20  TOTAL 


OBSERVED 


1  67 

60 

1  5 

1 

243 

49 

134 

1  35 

8 

326 

14 

75 

680 

1  78 

947 

4 

14 

183 

283 

484 

234 

283 

10,3 

470 

2000 

CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


OBSERVATION 


3  HR  PERSISTENCE 
FORECAST 


V? 

*  K.  * 

^  0-8 

8-12 

1  2-20 

>20 

TOTAL 

0-8 

1  59 

63 

1  7 

_ 

243 

8-12 

58 

140 

1 1 1 

1  7 

326 

12-20 

18 

106 

656 

167 

947 

5  I  19  167  293  484 


951 


CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


240 


328 


481 


2000 


Table  12.  OWS  Echo  Independent  Sample. 


6  HR  FORECAST 
FORECAST 

0-8  8-12  12-20  >20  TOTAL 

0-8 

8-12 

OBSERVED 

1  2-20 

>20 

TOTAL 


1  57 

66 

1 - 

23 

0 

246 

60 

1  09 

141 

22 

332 

33 

88 

500 

294 

915 

10 

23 

162 

307 

507 

260 

291 

826 

623 

2000 

a 


CORRECT  =  54  * 
SKILL  *  .330 
THREAT 

(0-8)  =  .449 
(8-12)  =  -212 
(12-20)  =  -402 
( >20)  «  -373 
Average  =  -370 


6  HR  PERSISTENCE 
FORECAST 

^  0-8  8-12  12-20  >20  TOTAL 

0-8 

3-1  2 

OBSERVATION 

1  2-20 
>20 
TOTAL 


145 

58 

34 

3 

— 

240 

46 

no 

1  27 

38 

321 

33 

1  1  5 

549 

227 

924 

5 

235 

241 

515 

229 

31  7 

945 

509 

2000 

b 

23 


CORRECT  3  5235 
SKILL  =  .295 
THREAT 

(0-8)  ■  -447 
(8-12)  *  -208 
(12-20)  =  -415 
( >20)  =  -307 

Average  a 


Table  13.  OWS  Echo  Independent  Sample. 


OBSERVED 


OBSERVATION 


9  HR  FORECAST 


FORECAST 


0-8 

8-1  2 

1  2-20 

>20 

TOTAL 

0-8 

1  66 

60 

33 

1 

260 

8-12 

86 

98 

1  1  7 

39 

340 

12-20 

48 

1  06 

394 

3  3 7 

885 

>20 

20 

32 

1  70 

293 

51  5 

TOTAL 

320 

296 

714 

670 

2000 

a 


CORRECT  =  48", 
SKILL  *  .261 
THREAT 

(0-8)  =  -400 
(8-12)  *  .182 
(12-20)  *  -596 
( >20)  =  -328 
Average  *  -422 


9  HR  PERSISTENCE 
FORECAST 


,  0-8 

CM 

r— 

1 

00 

12-20 

>20 

TOTAL 

0-8 

85 

48 

22 

6 

161 

3-12 

28 

68 

87 

31 

214 

1  2-20 

36 

65 

375 

1  77 

653 

>20 

7 

23 

198 

161 

389 

TOTAL 

1  56 

204 

682 

375 

1417 

b 


CORRECT 

=  49  S 

SKILL 

=  .235 

THREAT 

(0-8) 

=  .366 

(8-12) 

=  .194 

(12-20) 

=  .390 

( >20) 

=  .266 

Average 

*  .325 

24 


Table  14.  OWS  Echo  Independent  Sample. 


12  HR  FORECAST 
FORECAST 

0-3  8-12  12-20  >20  TOTAL 

0-8 

8-12 

OBSERVED 

12-20 
>20 

TOTAL 


1 2  HR  PERSISTENCE 
FORECAST 

0-8  8-12  12-20  >20  TOTAL 

0-8 

8-1  2 

OBSERVATION 

1  2-20 
>20 
TOTAL 


67 

36 

19 

6 

1  28 

23 

45 

63 

25 

1  56 

14 

59 

259 

1  27 

459 

1  1 

24 

1  26 

1  1  1 

272 

1  1  5 

1  64 

467 

269 

1015 

b 


25 


1  56 

57 

37 

2 

252 

1  06 

80 

103 

40 

329 

52 

1  34 

387 

326 

899 

23 

41 

184 

272 

520 

337 

31  2 

71  1 

640 

2000 

a 


-if.  I  ■HIC**"-""'*'  -' 


CORRECT 

SKILL 

THREAT 

(0-8) 
(3-12) 
(12-20) 
( >20) 
Average 


CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


Table  15.  OWS  Echo  Dependent  Sample. 


OBSERVED 


OBSERVATION 


3  HR  FORECAST 
FORECAST 


..V? 

^  0-8 

8-1  2 

1  2-20 

>20 

TOTAL 

0-8 

1  20 

89 

26 

1 

236 

8-12 

57 

165 

1  59 

11 

392 

12-20 

20 

94 

639 

1  33 

886 

>20 

1  2 

22 

209 

243 

486 

TOTAL 

209 

370 

_ 

1033 

388 

2000 

a 


CORRECT 

00 

LO 

II 

SKILL 

*  .383 

THREAT 

(0-8) 

*  .369 

(8-12) 

*  .276 

(12-20) 

=  .497 

( >20) 

*  .385 

Average 

=  .417 

3  HR  PERSISTENCE 
FORECAST 

^  0-8  8-12  12-20  >20  TOTAL 

0-8 

8-12 

12-20 
>20 
TOTAL 


1  24 

84 

18 

1  0 

236 

66 

1  70 

1  30 

26 

392 

27 

118 

561 

1  80 

886 

1  3 

23 

190 

260 

486 

230 

395 

899 

476 

2000 

b 


CORRECT  *  56" 
SKILL  *  .359 


THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


.362 
.275 
.458 
.370 
.  390 


S 


26 


Table  16.  OWS  Echo  Dependent  Sample. 


OBSERVED 


OBSERVATION 


6  HR  FORECAST 
FORECAST 


<  0-8 

8-1  2 

1  2-20 

>20 

TOTAL 

0-8 

1  1  6 

85 

43 

_ 

1 

245 

8-12 

79 

1  73 

1  50 

22 

424 

12-20 

35 

117 

533 

184 

869 

>20 

16 

35 

1  98 

21  3 

462 

TOTAL 

246 

410 

924 

420 

2000 

a 


CORRECT 

5 

52 

SKILL 

S 

.  303 

THREAT 

(0-8) 

s 

.309 

(8-12) 

s 

.262 

(12-20) 

s 

.423 

( >20) 

= 

.  318 

Average 

= 

.352 

6  HR  PERSISTENCE 
FORECAST 


,  N^O-8 

8-12 

1  2-20 

>20 

TOTAL 

0-8 

1  36 

78 

48 

11 

273 

CM 

1 

00 

66 

1  68 

1  51 

38 

423 

12-20 

49 

1  38 

485 

181 

853 

>20 

1  7 

37 

D 

21  5 

451 

TOTAL 

268 

421 

866 

445 

2000 

b 

27 


CORRECT 

=  50:S 

SKILL 

=  .291 

THREAT 

(0-8) 

=  .336 

(8-12) 

=  .243 

(12-20) 

=  .393 

( >20) 

=  .316 

Average 

=  .337 

Table  17.  OWS  Echo  Dependent  Sample. 


OBSERVED 


OBSERVATION 


9  HR  FORECAST 
FORECAST 

0-8  8-1  2  1  2-20  >20  TOTAL 

0-8 

8-12 
12-20 
>20 

TOTAL 


9  HR  PERSISTENCE 
FORECAST 

V^O-8  8-1  2  1  2-20  >20  TOTAL 

0-8 

8-1  2 

12-20 
>20 
TOTAL 


147 

72 

46 

1  7 

282 

64 

140 

155 

39 

398 

49 

144 

475 

215 

883 

1  3 

44 

205 

1  75 

437 

273 

400 

881 

446 

2000 

h 


no 

84 

44 

2 

240 

106 

148 

1  50 

28 

432 

45 

1  38 

484 

223 

890 

22 

27 

192 

197 

438 

283 

397 

870 

450 

2000 

a 


CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


CORRECT 

SKILL 

THREAT 

(0-8) 

(8-12) 

(12-20) 

(>Z0) 

Average 


OBSERVED 


OBSERVATION 


Table  18.  OWS  Echo  Dependent  Sample 


1  2  HR  FORECAST 
FORECAST 


v* 

0-8 

^  0-8 

8-12 

12-20 

>20 

TOTAL 

1  1  1 

71 

44 

7 

233 

8-12 

1  09 

1  36 

1  51 

26 

422 

1  2-20 

56 

1  34 

521 

202 

91  3 

>20 

26 

27 

189 

1  90 

432 

TOTAL 

302 

368 

905 

425 

2000 

a 


1  2  HR  PERSISTENCE 
FORECAST 


V.^0-8 

8-1  2 

1  2-20 

>20 

TOTAL 

0-8 

1  26 

68 

61 

1  5 

270 

8-12 

70 

1  34 

1  54 

42 

400 

12-20 

39 

139 

452 

221 

851 

>20 

28 

46 

21  1 

1  94 

479 

TOTAL 

263 

387 

878 

472 

2000 

b 


CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 


CORRECT 

SKILL 

THREAT 

(0-8) 
(8-12) 
(12-20) 
( >20) 
Average 
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Table  19.  Ocean  Weather  Station  Charlie. 


INDEPENDENT 

HEIDKE  SCORES 

DEPENDENT 

F 

P 

HR 

F 

P 

507 

.  483 

3 

.  455 

.  459 

429 

.  369 

6 

.  403 

.  356 

389 

.  303 

9 

.  342 

.  300 

343 

.  254 

1  2 

.  299 

.  267 

417 

.  352 

Average 

.  374 

.  345 

0036 

.0074 

a2 

.0035 

.0053 

1 

.144 

F-Val ue 

.291 

.32 

Probabi 1 i ty 

.61 

I 


THREAT  SCORES 

INDEPENDENT 

DEPENDENT 

F 

P 

HR. 

F 

P 

652 

.  621 

3 

.  547 

.  577 

560 

.  569 

6 

.  519 

.  505 

553 

.  518 

9 

.  490 

.  465 

552 

.  482 

1  2 

.  472 

.  450 

579 

.  547 

Average 

.  507 

.  499 

0018 

.0027 

o2 

.0008 

.0024 

.667 

F-Val ue 

.055 

.44 

Probabi 1 i ty 

.82 

a 
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Table  19.  Ocean  Weather  Station  Echo. 


HEIDKE  SCORES 


INDEPENDENT  DEPENDENT 


F 

P 

HR 

F 

P 

447 

.  443 

3 

.  383 

.  359 

330 

.  295 

6 

.  303 

.  291 

261 

.  235 

9 

.  239 

.  238 

222 

.  230 

1  2 

.  246 

.  219 

31  5 

.  300 

Average 

.  292 

.  276 

0073 

.0074 

a2 

.0033 

.0029 

,0414 

F- Va 1 ue 

,1222 

84 

Proba  bi 1 i ty 

74 

THREAT  SCORES 


INDEPENDENT 

F  P 

HR 

DEPENDENT 

F  P 

— 

— 

— 

403 

.  459 

3 

.  41  7 

.  390 

370 

.  358 

6 

.  352 

.  337 

422 

.  325 

9 

.  309 

.  309 

291 

.  31  7 

1  2 

.  320 

.  295 

371 

.364 

Average 

.  349 

.  332 

0025 

.0032 

2 

a 

.001  7 

.001  3 

,0239 

F-Val ue 

2724 

,88 

Pro  bab i 1 i ty 

62 

b 
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Table  20 


Ocean  Weather  Station  Charlie 
PERCENT  CORRECT 

I NDEPENDENT  DEPENDENT 


F 

P 

HR 

F 

P 

77 

75 

3 

70 

71 

70 

69 

6 

67 

65 

70 

66 

9 

65 

61 

69 

62 

12 

63 

60 

a 

Ocean  Weather 

■  Statior 

i  Echo 

INDEPENDENT 

DEPENDENT 

F 

P 

HI 

F 

P 

63 

62 

3 

58 

56 

54 

52 

6 

52 

50 

48 

49 

9 

47 

47 

45 

47 

12 

48 

45 

b 
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APPENDIX 

THREAT  AND  SKILL  SCORES  DEFINITIONS 


THREAT  SCORES 

Consider  a  2x2  contingency  table. 

Fo  re  ca  s  t 

F^  F^  Totals 

Observed 


Total  s 


The  threat  score  is  defined  as 


Tb  =  d-j  b+c  • 

This  score  examines  the  ratio  of  correct  forecasts  (d 
the  sum  of  correct  and  incorrect  (d+b+c).  If  b=c=0,  then 
Thereto  re  0 <_T S  <_1  . 

Notice  that  since  R-pd-j+b  and  c-pd-j+c, 

dl 

TC  -  • 


Using  this  formula,  threat  scores  for  nxn  contingency 
can  be  computed  for  each  diagonal  element  ( d  ^  ,  i=l,...,n). 
(nxn)  contingency  table,  n  threat  scores  are  computed. 

To  compare  threat  scores  for  two  contingency  tables, 
weighted  average  is  computed. 


1  )  to 
TS  =  1  . 


tables 
For  a 

a 
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Then  (TS)^,  the  weighted  average,  is  defined  by 


R-i  +c-j  RolCo 

(ts)a  =  L)  (TSh  +  (-nr1)  (ts)2. 

SKILL  SCORES 

Heidke  skill  scores  are  defined  for  (2x2)  contingency  tabl 
as  follows. 


Fj  Total 

°1 

Observations  0  2 

Total 

,  .  X-C 

s  '  rrc 

where 

X  =  d 1 +d  2 

,  («,C,)  _  (R2C2) 

C - T -  T - 

For  a  (nxn)  contingency  table  the  generalized  formula  is 


HI 

El 

Hi 

■a 

The  Heidke  skill  score  compares  total  correct  forecasts  with 
the  expected  total  correct  forecasts,  as  determined  from  a  chance 


selection  procedure. 

For  example,  the  chance  that  any  single  forecast  event  will 

C1 

fall  in  the  first  column  is  (*— )  .  The  chance  that  any  single 

R1 

observed  event  will  fall  in  the  first  row  is  (— )  .  Therefore, 
the  chance  that  any  single  foreca s t-obser va ti o n  event  will  fall  in 


'1 


the  first  diagonal  cell  (cell  d-|  )  is  (y-1-)  (— )  ,  and  the  expected 
number  of  events  to  fall  in  cell  d^  is  the 


C,  Ri  C ,  R , 

(y-  )x(~)  (T)  =  (— j — 


Therefore 


1 


n 


C  -  j  Z  (  C ,  R ,  ) 
1  i  =  1  11 


is  simply  the  total  expected  number  of  events  to  fall  on  the 
di agonal  el ements  . 

The  skill  score  then  is  the  ratio  of  actual  minus  expected 
to  total  minus  expected. 

If  x  =  c  ,  than  s=Q,  and  thus  the  forecast  method  shows  no  skill 
better  than  random  chance.  If  X=T,  S=1 ;  the  forecast  is  perfect. 
(Notice  it  is  possible  to  get  negative  skills  (i.e.,  X<C),  which 
means  that  chance  beats  the  forecast  method.) 
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